If self.sampler.sample returns a generator or iterator (which is common for streaming or lazy samplers), the check if responses will evaluate to True because generator/iterator objects are always truthy in Python, even if they yield no elements. Subsequently, calling list(responses)[0] on an empty generator will raise an IndexError: list index out of range.

To prevent this, convert responses to a list first, and then check if the list is non-empty before accessing the first element.

In llm_backup_async, calling the synchronous network-bound function _call_teacher directly will block the entire asyncio event loop for the duration of the API call (which can take several seconds). This defeats the purpose of using asynchronous programming.

Use asyncio.to_thread to run the synchronous _call_teacher function in a separate thread, preventing it from blocking the event loop.

The current implementation of _build_key and _extract_param manually matches arguments by index and checks kwargs. This approach is fragile and does not handle default values correctly when parameters are omitted from the call (relying on their defaults). For example, if a parameter in key_params has a default value (like query=None) and is omitted from the call, it won't be included in the key, leading to different keys for semantically identical calls.

Using inspect.signature's bind and apply_defaults provides a robust, standard way to handle positional/keyword arguments and default values.

def _build_key(sig: inspect.Signature, func_name: str, args: tuple, kwargs: dict, key_params: Sequence[str]) -> str:
    """Build a unique key from specified parameter values using signature binding."""
    try:
        bound = sig.bind(*args, **kwargs)
        bound.apply_defaults()
        arguments = bound.arguments
    except TypeError:
        arguments = kwargs

    key_parts = [func_name]
    for name in key_params:
        if name in arguments:
            key_parts.append(f"{name}={_serialize_value(arguments[name])}")
    raw_key = "|".join(key_parts)
    return hashlib.md5(raw_key.encode()).hexdigest()


def _serialize_value(value: Any) -> str:
    try:
        return json.dumps(value, sort_keys=True, default=str)
    except (TypeError, ValueError):
        return str(value)


def _extract_param(sig: inspect.Signature, args: tuple, kwargs: dict, name: str) -> Any:
    """Extract a named parameter from args/kwargs given the signature."""
    try:
        bound = sig.bind(*args, **kwargs)
        bound.apply_defaults()
        return bound.arguments.get(name)
    except TypeError:
        return kwargs.get(name)

Update the call sites in llm_backup to pass the pre-computed sig object to _build_key and _extract_param for robust signature-based argument binding.

Update the call sites in llm_backup_async to pass the pre-computed sig object to _build_key and _extract_param for robust signature-based argument binding.